On Autistic Interpretations of Occam’s Razor: Explanatory and Creative Alternatives to the MDL principle
نویسندگان
چکیده
The Minimum Description Length (MDL) principle is the modern formalisation of Occam’s razor. It has been extensively and successfully used in machine learning (ML), especially for noisy and long sources of data. However, the MDL principle presents some paradoxes and inconveniences. After discussing these all, we address two of the most relevant: lack of explanation and lack of creativity. We present new alternatives to address these problems. The first one, intensional complexity, avoids extensional parts in a description, so distributing compression ratio in a more even way than the MDL principle. The second one, information gain, forces that the hypothesis is informative (or computationally hard to discover) wrt. the evidence, so giving a formal definition of what is to discover.
منابع مشابه
JOSÉ HERNÁNDEZ-ORALLO and ISMAEL GARCÍA-VAREA EXPLANATORY AND CREATIVE ALTERNATIVES TO THE MDL PRINCIPLE
The Minimum Description Length (MDL) principle is the modern formalisation of Occam’s razor. It has been extensively and successfully used in machine learning (ML), especially for noisy and long sources of data. However, the MDL principle presents some paradoxes and inconveniences. After discussing all these, we address two of the most relevant: lack of explanation and lack of creativity. We pr...
متن کاملComplexity Approximation Principle
We propose a new inductive principle, which we call the complexity approximation principle (CAP). This principle is a natural generalization of Rissanen’s minimum description length (MDL) principle and Wallace’s minimum message length (MML) principle and is based on the notion of predictive complexity, a recent generalization of Kolmogorov complexity. Like the MDL principle, CAP can be regarded...
متن کاملModel Selection with Informative Normalized Maximum Likelihood: Data Prior and Model Prior
Model selection has, in the last decade, undergone rapid growth for evaluating models of cognitive processes, ever since its introduction to the mathematical/cognitive psychology community (Myung, Forster, & Browne, 2000). The term “model selection” refers to the task of selecting, among several competing alternatives, the “best” statistical model given experimental data. To avoid ambiguity, “b...
متن کاملA Geometric Formulation of Occam’s Razor For Inference of Parametric Distributions
I define a natural measure of the complexity of a parametric distribution relative to a given true distribution called the razor of a model family. The Minimum Description Length principle (MDL) and Bayesian inference are shown to give empirical approximations of the razor via an analysis that significantly extends existing results on the asymptotics of Bayesian model selection. I treat paramet...
متن کاملChapter 7 Asymptotics and Coding Theory : One of the n ! • Dimensions of Terry
Terry joined the Berkeley Statistics faculty in the summer of 1987 after being the statistics head of CSIRO in Australia. His office was just down the hallway from mine on the third floor of Evans. I was beginning my third year at Berkeley then and I remember talking to him in the hallway after a talk that he gave on information theory and the Minimum Description Length (MDL) Principle of Rissa...
متن کامل